Multiple hypothesis testing using the excess discovery count and alpha-investing rules

We propose an adaptive, sequential methodology for testing multiple hypotheses. Our methodology consists of a new criterion, the excess discovery count (EDC), and a new class of testing procedures that we call alpha-investing rules. The excess discovery count is the difference between the number of correctly rejected null hypotheses and a fraction of the total number of rejected hypotheses. EDC shares many properties with the false discovery rate (FDR), but is adapted to testing a sequence of hypotheses rather than a fixed set. Because EDC controls the count of incorrectly rejected hypotheses rather than a ratio, we are able to prove that a wide class of testing procedures that we call alpha-investing rules control EDC. Alpha-investing rules mimic alpha-spending rules used in sequential trials, but possess a key difference. When a test rejects a null hypothesis, alpha-investing rules earn additional probability toward testing subsequent hypotheses. Alpha-investing rules allow one to incorporate domain knowledge into the testing procedure and improve the power of the tests.